Towards Thorough Empirical Methods for AI Planning

Empirical investigations in the area of AI planning have been focused on a comparatively small set of benchmark tasks. Trying to design larger scale experiments for wellfounded empirical reasoning in that area, one encounters a number of severe problems. While some of these problems are inherent to the field, others have plainly been ignored. In our own work, we have made some first steps towards addressing these problems. The field of domain independent planning is concerned with developing problem solving techniques that are general in the sense that they can—ideally—be used for any application one wants to deal with. Many approaches have been proposed for modeling planning problems, i.e., for formal planning frameworks [Fikes and Nilsson, 1971; Pednault, 1989], as well as for general problem solving strategies within such frameworks [Penberthy and Weld, 1992; Blum and Furst, 1997; Bonet and Geffner, 2001]. For the evaluation of new planning strategies most publications in the area refer to a small set of benchmark planning tasks and make claims on the basis of roughly comparing performance on those tasks. While performance results on a few examples can give a general impression of the usefulness of an approach, a more thorough empirical evaluation is certainly desirable. When we tried to to design and interpret large scale experiments [Hoffmann and Nebel, 2001; Hoffmann, 2001], we encountered a number of problems which roughly fall into the following three categories. 1. Which planning domains should one choose? 2. Which examples should one choose within a domain? 3. What is an adequate formal way of interpreting the experimental data? Georges-Köhler-Allee, Geb. 52, 79110 Freiburg, Germany, Phone: +49 (761) 203-8229, Fax: +49 (761) 203-8222, WWW: http://www.informatik.uni-freiburg.de/˜ hoffmann As for problem 1, it is inherent in the field, and there does not seem to be much one can do about it. One could, of course, try to generate problem instances completely randomly without any reference to an application domain, similar to what has been done in the area of SAT algorithms [Mitchell et al., 1992]. However, we do not know of any method for generating random planning instances. Furthermore, even if such a method were known, it would be completely unclear whether the generated instances would be representative of the envisioned application. For these reasons, one usually starts with some planning domains such as block stacking, logistics, etc. However, it is not possible to obtain a set of domains representative for all applications one could think of. So one has to choose some domains arbitrarily. In our experiments, we settled for a collection of 20 domains that are justified in the sense that they are (amongst) the most frequently used domains within the planning community. As for problem 2, this is subdivided into three issues which all have been hardly addressed in the planning literature. Firstly, for almost all benchmark domains there is no definition specifying which planning instances belong to that domain. Secondly, for most domains there are some example instances publicly available, but rarely ever has someone published something about how random instances can be or should be generated. Thirdly, for almost all domains there is no notion of which instances are interesting. We have dealt with the first point by abstracting from the examples that are publicly available, largely following (yet unpublished) work done by Malte Helmert. Based on that, we have dealt with the second point by choosing and implementing for all our 20 domains the intuitively most obvious randomization strategy. That said, it is clear that we have not (yet) dealt with the third point at all. We have made our random generators publicly available along with descriptions of the randomization strategies. 1 This provides at least a starting The descriptions and the source code of all generators are available via http://www.informatik.unifreiburg.de/˜ hoffmann/ff-domains.html